Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow using Pkg protocol to get packages, artifacts & registries #1444

Merged
merged 1 commit into from
Nov 28, 2019

Conversation

StefanKarpinski
Copy link
Member

@StefanKarpinski StefanKarpinski commented Oct 12, 2019

First steps towards implementing #1377.

@StefanKarpinski StefanKarpinski changed the title use Pkg protocol to get packages, artifacts & registries [WIP] use Pkg protocol to get packages, artifacts & registries Oct 12, 2019
@codecov
Copy link

codecov bot commented Oct 12, 2019

Codecov Report

Merging #1444 into master will decrease coverage by 0.47%.
The diff coverage is 50%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1444      +/-   ##
==========================================
- Coverage   87.29%   86.82%   -0.48%     
==========================================
  Files          25       25              
  Lines        5406     5448      +42     
==========================================
+ Hits         4719     4730      +11     
- Misses        687      718      +31
Impacted Files Coverage Δ
src/PlatformEngines.jl 76.19% <100%> (ø) ⬆️
src/Artifacts.jl 90.14% <25%> (-0.97%) ⬇️
src/Types.jl 79.73% <40.74%> (-3.57%) ⬇️
src/Pkg.jl 78.57% <50%> (-11.43%) ⬇️
src/Operations.jl 93.02% <76.47%> (-0.35%) ⬇️
src/backwards_compatible_isolation.jl 84.7% <0%> (+1.02%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f97b18...ba75cc4. Read the comment docs.

src/Artifacts.jl Show resolved Hide resolved
src/Operations.jl Show resolved Hide resolved
@StefanKarpinski StefanKarpinski changed the title [WIP] use Pkg protocol to get packages, artifacts & registries use Pkg protocol to get packages, artifacts & registries Oct 29, 2019
@StefanKarpinski
Copy link
Member Author

Ok, this is no longer WIP: it works and https://pkg.julialang.org is currently working as a Pkg Server. Any thoughts on what the next steps here ought to be? This doesn't currently have tests, which I don't love, but it's also a bit hard to test locally and we weren't really explicitly testing installing packages from GitHub previously anyway, so maybe the implicit testing of installing packages is sufficient. I'd like to merge this to master on Pkg and then bump the version of Pkg that Julia master is using so that we get people who use Julia master to test this out.

@StefanKarpinski
Copy link
Member Author

Oh, the main drawback that this currently has is that it does not do diffs for updates, which means that when you update the registry, you have to download a whole new compressed registry tarball. I'm working on diffs, but it's a bit tricky, so I'd like to get this out there into people's hands first.

@fredrikekre
Copy link
Member

MethodError: no method matching pkg_server_registry_url(::Nothing) on CI.

Ok, this is no longer WIP: it works and https://pkg.julialang.org is currently working as a Pkg Server.

Seems to be somewhat out of sync with the new releases to the registry, couldn't install the most recent version of some packages.

Is the server spec documented somewhere? Is it basically following #1377 (comment) ?

@fredrikekre
Copy link
Member

fredrikekre commented Nov 1, 2019

which means that when you update the registry, you have to download a whole new compressed registry tarball.

This does not seem to happen? Since you just get a snapshot of the registry I don't think it will be updated, since we only update registries that are git repos.

Edit: Nvm, Just that pkg.julialang.org does not sync with JuliaRegistries/General.

@fredrikekre
Copy link
Member

Lots of

Warning: could not download https://pkg.julialang.org/registries

so something seems to be off. Maybe because we do some trickery in order to not update the registry all the time on CI? E.g. in

function isolate(fn::Function; kwargs...)
and
function temp_pkg_dir(fn::Function;rm=true)

@StefanKarpinski
Copy link
Member Author

Any idea why the CI process would be unable to reach https://pkg.julialang.org/registries?

@StefanKarpinski StefanKarpinski changed the title use Pkg protocol to get packages, artifacts & registries allow using Pkg protocol to get packages, artifacts & registries Nov 8, 2019
@StefanKarpinski
Copy link
Member Author

I've made it opt-in now by setting JULIA_PKG_SERVER=pkg.julialang.org in the environment. Given that it's opt in and works locally and the server is correctly updating now, I think that we can probably merge this if CI passes. There will probably be a few changes before 1.4 feature freeze, including the ability to update things with diffs, and the ability to inject authentication headers, but it would be good to get some testing of this on master, even opted out, in the mean time.

@staticfloat
Copy link
Member

I've attempted to fix the merge conflict, if it goes through, let's merge.

src/Types.jl Outdated Show resolved Hide resolved
src/Artifacts.jl Show resolved Hide resolved
src/PlatformEngines.jl Show resolved Hide resolved
@StefanKarpinski StefanKarpinski merged commit 258d74e into master Nov 28, 2019
@StefanKarpinski StefanKarpinski deleted the sk/protocol branch November 28, 2019 13:09
@KristofferC
Copy link
Member

success = install_archive(urls[pkg.uuid], hashes[pkg.uuid], path)

needs updating to the new signature (https://ci.appveyor.com/project/ararslan/primes-jl/builds/29199679/job/6ff0p4f64rxbqk6b)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants